Image sharpening technique

ABSTRACT

A sharpening technique for digital images for presentation on a display.

BACKGROUND OF THE INVENTION

The present invention relates to sharpening of digital images and videofor presentation on a display.

The resolution of a typical liquid crystal display is approximately720×480 which is on the order of standard definition television. Theresolution of a typical very high resolution diagonal display is4096×2160 and typically has viewing distances that are less than 2picture heights. In order to display an image on a display in aaesthetically pleasing manner sharpening of the image is typicallyperformed. The lack of sharpness typically results from imperfections inthe captures process and/or loss of details due to image compression.However, sharpening of the image tends to result in annoying artifacts.

The traditional approaches to sharpening, such as using an unsharp mask,generally result in the introduction of visible degradation into theimage being sharpened. The visual degradation is primarily due toenhancing the high frequency components and noise that often dominatesthe high frequency bands of an image. To moderate the enhancement of thevisible degradation of the image, the amount of sharpening performed iscontrolled.

It is desirable to increase the sharpness of the image content while atthe same time suppressing the noise in the resulting image.

The foregoing and other objectives, features, and advantages of theinvention will be more readily understood upon consideration of thefollowing detailed description of the invention, taken in conjunctionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates image sharpening without resolution change.

FIG. 2 illustrates image sharpening with resolution change.

FIG. 3 illustrates another image sharpening with resolution change.

FIG. 4 illustrates another image sharpening with resolution change.

FIG. 5 illustrates another image sharpening with resolution change.

FIG. 6 illustrates another image sharpening with resolution change.

FIG. 7 illustrates bilateral filter with pre-filtering.

FIG. 8 illustrates comfort noise/texture equalization for noisesuppression.

FIG. 9 illustrates another image sharpening with resolution change.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENT

Image sharpening techniques are designed to enhance the high frequencyaspects of the image. High frequency aspects, such as edges around majorfeatures of the image, are particularly desirable to sharpen in order toimprove the visual appearance of the image to the viewer. Howeversharpening high frequency aspects, such as noise in the texture regionsof the image, are not desirable because it degrades the visualappearance of the image to the viewer. These tradeoff are generallycontradictory to one another, while on the one hand a desire to enhancethe high frequency aspects of portions of the image while on the otherhand a desire not to enhance the high frequency aspects of otherportions of the image.

Attempting to develop a universal sharpening filter has proved to beelusive. It has been determined that sharpening an image is best doneusing multiple different filters or otherwise a filter applied to only aportion of the image. Each of the different filters should filter theimage in a different manner so that the beneficial aspects of each ofthe filters may be taken advantage of. It is to be understood that anyfilter maybe used, such as those that sharpen the image.

Referring to FIG. 1, it was determined that the system may preprocessthe incoming image with a non-linear edge-adaptive smoothing filter 100that is used to separate the image into multiple “channels”. It is notedthat these “channels” need not be frequency bands, or directly relatedto the frequency domain. The filter may be referred to as anedge-adaptive smoothing filter or any other suitable filter. One channel102 may contain the edge content and slowly varying “flat” regions ofthe image where edges (which generally contain high spatial frequencies)are retained and remain generally sharp, but textures (which alsocontain high spatial frequencies) are generally filtered out. The otherchannel 104 may contain the textures of the image minus edges. A linearfilter is not especially suitable for such selective filtering.

These two channels, namely, texture information (T) 104 and edge+flatregions 102 (E), can be used in a number of ways to produce a highquality sharpened image. One approach is to use a sharpening technique106 on the edge+flat regions 102 to enhance the image. The edge+flatregions 102 is generally noise free so the enhancement of that channeldoes not substantially increase the resulting noise in the final image.A noise suppression technique 108 on the texture channel reduces 104 thenoise that is contained in that channel. Thereafter, the two channels T104 and E 102 are recombined 110 into the final output 112. Accordingly,those edge aspects of the image that are desirable to sharpen aresuitably enhanced without substantial enhancement of the noise, whilethe other aspects of the image that are not desirable to sharpen aresuitably attenuated in a manner to reduce the noise.

When the input resolution and the desired output resolutions aredifferent, one or more scaling operations may be included to re-scalethe image. Referring to FIG. 2, one technique for scaling the image isto include an image scaling operation 200 at the output thereof. In thismanner, the image processing pipeline may process the image at a firstinput resolution then use a high quality scaling operation 200 toachieve the desired output results. This manner of processing maintainsthe image processing pipeline with relatively low complexity

Referring to FIG. 3, another technique for scaling the image is toinclude an image scaling operation 210 at the input thereof. In thismanner, the image processing pipeline may be designed to process imagesat a single image resolution that is the same as the scale provided atthe output of the input image scaling operation, which decreases thecomplexity of the image processing pipeline. The image processingpipeline may be optimized for a single image scale.

Referring to FIG. 4, another technique for scaling the image is toinclude an image scaling operation 220 after the noise suppression inthe texture channel (T). The texture channel has particular imagecharacteristics and the scaling filter may be designed to accommodatethe particular image characteristics. The combination of the scalingoperation and noise suppression may be such that improved imagecharacteristics are achieved. An image scaling operation 240 after thesharpening in the flat images+edges channel (E). The channel E hasparticular image characteristics and the scaling filter may be designedto accommodate the particular image characteristics. The combination ofthe scaling operation and sharpening may be such that improved imagecharacteristics are achieved.

Referring to FIG. 5, another technique for scaling the image is toinclude an image scaling operation 230 before the noise suppression inthe texture channel (T). The scaling may be designed to accommodate theadded noise in the channel, that is thereafter suppressed by the noisesuppression. The combination of the scaling operation and noisesuppression may be such that improved image characteristics areachieved. An image scaling operation 250 before the sharpening in theflat images+edge channel (E). The scaling may be designed to accommodatethe reduced noise in the channel. The combination of the scalingoperation and sharpening may be such that improved image characteristicsare achieved.

Referring to FIG. 6, depending on the particular implementationrequirements, one or more of the scaling operations may be includedtogether with the system. The combination of the different scalingoperations achieves a robust design that is readily adaptable todifferent images and system requirements. It is to be understood thateach of the scaling operations may be the same or different. By way ofexample, the scaling 230 may be a bilinear upscaling operation whereasscaling operation 250 may be a non-linear scaling operation.

More specifically, the edge adaptive smoothing filter 100 applies signaldependent processing to separate strong object edges from fine textureand noises. As one specific example, the system may use bi-lateralfilters. One such bilateral filter may be formulated as

${{\hat{x}\lbrack k\rbrack} = \frac{\sum\limits_{n = {- N}}^{N}{{W\left\lbrack {k,n} \right\rbrack}{x\left\lbrack {k - n} \right\rbrack}}}{\sum\limits_{n = {- N}}^{N}{W\left\lbrack {k,n} \right\rbrack}}},$which consists of a weight calculation step and a weighted averagingstep. For notational simplicity one may assume a one-dimensional filterbut there is no loss of generality in doing that. Here x is the inputsignal and {circumflex over (x)} is the filtered signal and the filterkernel W consists of a spatial kernel W_(s), and a range kernel W_(R):W[k,n]=W_(s)[k,n]·W_(R)[k,n]. Boxcar, gaussian and triangular kernelsmay likewise be used for kernels. Other techniques, such as anisotropicdiffusion and adaptive smoothing may likewise be used. In general, anyfilter that tends to preserve edge features while reducing noise may beused.

The kernel calculation in bilateral filters is signal dependent andserves to differentiate strong object edges from texture/noise/artifact.This is achieved by assigning lower weights to pixels in the filterwindow whose intensity values are far from the center pixel intensity.Preferably, a pre-filtering step f(x) before the weight calculation isincorporated to achieve better noise and artifact robustness. Byreducing the noises and artifacts, the system may improve the weightcalculation which in turn improves the filtered outcome. This isillustrated as a block diagram in FIG. 7. Several choices of f may beused, including for example, linear low pass filters and median filters.Other nonlinear smoothing filters can also be applied.

It is desirable to suppress noise in the texture channel. Since thetexture channel primarily contains all the noise and artifacts,attenuating the texture channel will suppress the amount of noise andartifacts in the final output. This can achieved by applying a weight ωto the texture channel, and the weighting can be either spatiallyinvariant or spatially variant. In the case of spatially variant texturechannel weighting, the information from the bilateral filter can beleveraged to determine the amount of suppression. More specifically anedge measure

${E\lbrack k\rbrack} = \frac{W\left\lbrack {k,0} \right\rbrack}{\sum\limits_{n = {- N}}^{N}{W\left\lbrack {k,n} \right\rbrack}}$can be derived from the bilateral filter, with higher E valuesindicating that the current pixel is closer to an edge. Subsequently thetexture channel weights can be derived by ω[k]=(1−E[k])*α+β. Here α,βare introduced to control the range and mean for the texture channelweighing.

It is desirable to suppress coding artifacts in the texture channel.Most of the displayed contents are compressed one way or another, andinevitably contain various compression artifacts. Among those the mostvisually objectionable is the blocking artifacts due to thenon-overlapping block transforms used in most existing compressionalgorithms. The blocking artifact manifest itself as small square blocksall over the decompressed images, and most noticeably in flat regions. Adeblocking filter may be applied on the texture channel to suppressblocking artifacts. The main difference between the texture channeldeblocking and a conventional deblocking is that the texture channel isvery noisy and the real object edges are not present or not as strong.One may apply the following steps for the deblocking of the texturechannel:

First, calculate an edge measure for each pixel by using

${E\lbrack k\rbrack} = \frac{W\left\lbrack {k,0} \right\rbrack}{\sum\limits_{n = {- N}}^{N}{W\left\lbrack {k,n} \right\rbrack}}$and an edge measure for a block by summing up the per-pixel measure.

Second, in the texture channel deblocking process, control the amount offiltering per blocking edge using the edge measure calculated in theprevious step. The smaller the edge measure, the stronger the filter is.

It is desirable to modify the image to provide a sufficient amount ofhigh frequency content so as to have a pleasing appearance. Attenuatingthe texture channel reduces the noise at the expense of also removingfine detail. This may reduce the perceived sharpness to a humanobserver, as the resulting output image has less high frequencyinformation than expected. To overcome this issue, texture synthesis maybe added to the image to mask structured, coding noise artifacts withoutreducing high frequency information. A block diagram for the comfortnoise is shown in FIG. 8. To be clear, FIG. 8 provides a more detaileddescription of the noise suppression block shown in FIG. 1.

As can be seen from FIG. 8, the noise suppression process takes thetexture channel as input. The texture data is then analyzed by an imageanalysis process. In one embodiment, the image analyzer extracts thevariance and correlation of image regions within the texture channel. Ofcourse, other statistics may be extracted. For example, the location ofedges may be identified. By way of example, the mean value of the imageregion may be extracted.

Outputs from the image analysis block are used to control filtering andmultiplication operations. For example, the correlation estimatecontrols the linear filter shown in FIG. 8. Additionally, the noisepower estimate controls the multiplication operator that follows thelinear filter output. Furthermore, the mean and edge information is usedto control both the multiplication following the linear filter as wellas the multiplication of the input data. Finally, the scaled inputtexture channel and the modified random noise are added and provided asoutput of the noise suppression component.

In many cases, the non-linear splitting filter will not exactly separatethe edges from the textures. Accordingly, some information will leakinto each channel from the other channel. Consequently, the adding backtogether of the two channels may be based upon sensed imagecharacteristics to achieve the final image.

Referring to FIG. 9, a more generalized framework may be used for animproved scaling technique. The technique includes a multi-channelspatial decomposition which may decompose the image into any suitableset of channels. The decomposition may be linear or non-linear. Inaddition, the spatial decomposition may be more than two channels, ifdesired. The channels may include characteristics, such as for example,graphics versus natural, text versus non-text, face versus non-face,texture, and edges+flat region, scrolling text, film grain noise.Depending on the particular implementation, the input image for one ormore channels may not be filtered but rather scaled in any manner. Thechannel specific scaling may use any suitable set of filters or omitted.The channel specific noise reduction may use any suitable set of filtersor omitted. If desired, the filters may also include temporalinformation of a video when scaling a particular image. Blendingtogether of the separate channels may be done using any suitabletechnique.

The terms and expressions which have been employed in the foregoingspecification are used therein as terms of description and not oflimitation, and there is no intention, in the use of such terms andexpressions, of excluding equivalents of the features shown anddescribed or portions thereof, it being recognized that the scope of theinvention is defined and limited only by the claims which follow.

1. A method for modifying an image from a first sharpness to a modifiedimage having a different sharpness: (a) receiving said image having saidfirst sharpness; (b) filtering said image with an image filter to resultin the creation of a first channel and a second channel, wherein saidfirst channel includes primarily texture information and said secondchannel includes primarily edge and flat region information; (c)filtering said first channel to attenuate higher frequency content ofsaid image in said first channel; (d) filtering said second channel tosharpen said image in said second channel; (e) combining said firstchannel and said second channel to form said modified image having saiddifferent sharpness; and (f) where said filtered image satisfies atleast one of the following criteria: (1) as a result of step (c), saidfiltered image is at least one of previously and subsequently scaledwith a scaling filter on the image in a first channel; and (2) as aresult of step (d), said filtered image is at least one of previouslyand subsequently scaled with a scaling filter on the image in a secondchannel.
 2. The method of claim 1 wherein said image filter is anon-linear filter.
 3. The method of claim 2 wherein said non-linearfilter is an adaptive filter.
 4. The method of claim 3 wherein saidadaptive filter is edge adaptive.
 5. The method of claim 4 wherein saidedge adaptive filter includes a smoothing function.
 6. The method ofclaim 1 wherein said filtering further creates a third channel.
 7. Themethod of claim 6 wherein said filtering further creates a fourthchannel.
 8. The method of claim 7 wherein said filtering further createsa fifth channel.
 9. The method of claim 8 wherein said filtering furthercreates a sixth channel.
 10. The method of claim 1 wherein said modifiedimage is subsequently scaled with a scaling filter.
 11. The method ofclaim 1 wherein said image is previously scaled with a scaling filterprior to being filtered with said image filter.
 12. The method of claim1 wherein said filtered image as a result of step (c) is previouslyscaled with a scaling filter on the image in said first channel.
 13. Themethod of claim 12 wherein said filtered result of step (d) is scaledwith a scaling filter on the image in said second channel.
 14. Themethod of claim 13 wherein said scaling filter of said first channel andsaid scaling filter of said second channel are different filters. 15.The method of claim 1 wherein said filtered image as a result of step(c) is subsequently scaled with a scaling filter in said first channel.16. The method of claim 15 wherein said filtered result of step (d) isscaled with a scaling filter on the image in said second channel. 17.The method of claim 16 wherein said scaling filter of said first channeland said scaling filter of said second channel are different filters.18. The method of claim 1 wherein said filtered image as a result ofstep (d) is previously scaled with a scaling filter on the image in saidsecond channel.
 19. The method of claim 18 wherein said filtered resultof step (c) is scaled with a scaling filter on the image in said firstchannel.
 20. The method of claim 19 wherein said scaling filter of saidfirst channel and said scaling filter of said second channel aredifferent filters.
 21. The method of claim 1 wherein said filtered imageas a result of step (d) is subsequently scaled with a scaling filter insaid second channel.
 22. The method of claim 21 wherein said filteredresult of step (c) is scaled with a scaling filter on the image in saidfirst channel.
 23. The method of claim 22 wherein said scaling filter ofsaid first channel and said scaling filter of said second channel aredifferent filters.
 24. The method of claim 1 further comprising usingpre-filters to reduce noise.
 25. The method of claim 1 furthercomprising filtering in step (c) is based upon information obtained fromstep (b).
 26. The method of claim 25 wherein an edge measure iscalculated from step (b).
 27. The method of claim 26 wherein said edgemeasure is used as the basis to attenuate said first channel.
 28. Themethod of claim 1 further comprising adding synthesized high frequencycontent to said modified image.
 29. A method for modifying an image froma first sharpness to a modified image having a different sharpness: (a)receiving said image having said first sharpness; (b) filtering saidimage with an image filter to results in the creation of a first channeland a second channel, wherein said first channel includes primarilytexture information and said second channel includes primarily edge andflat region information; (c) filtering said first channel to attenuatehigher frequency content of said image in said first channel; (d)filtering said second channel to sharpen said image in said secondchannel; (e) combining said first channel and said second channel toform said modified image having said different sharpness; and (f) usingpre-filters to reduce compression artifacts in said image.
 30. A methodfor modifying an image from a first sharpness to a modified image havinga different sharpness: (a) receiving said image having said firstsharpness; (b) filtering said image with an image filter to results inthe creation of a first channel and a second channel, wherein said firstchannel includes primarily texture information and said second channelincludes primarily edge and flat region information; (c) filtering saidfirst channel to attenuate higher frequency content of said image insaid first channel; (d) filtering said second channel to sharpen saidimage in said second channel; (e) combining said first channel and saidsecond channel to form said modified image having said differentsharpness; and (f) using at least one pre-filter to reduce at least oneof compression artifacts and noise, wherein said at least one pre-filteris performed prior to a weight calculation included with said imagefilter.